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Simultaneous Segmentation of Multiple or Composed Objects by Mesh Adaptation 



5 DESCRIPTION 

The present invention relates to a method for the simultaneous 
segmentation of multiple or composed objects in an image. Furthermore, the present 
invention relates to an image processing device adapted for executing the method for 
the simultaneous segmentation of multiple or composed objects in an image and a 

10 computer program comprising instruction code for executing the method for the 
simultaneous segmentation of multiple or composed objects. 

Segmentation methods are used to derive geometric models of, for 
example, organs or bones from volumetric image data, such as CT, MR or US images. 
Such geometric models are required for a variety of medical applications, or generally in 

15 the field of pattern recognition. For medical or clinical applications, an important 

example is cardiac diagnosis, where geometric models of the ventricles and the myocard 
of the heart are required for perfusion analysis, wall motion analysis and computation of 
the ejection fraction. Another important clinical application is radio-therapy planning, 
where the segmentation of multiple organs and bones in, for example, the prostrate 

20 region (femur heads, rectum, prostate, bladder) is necessary for the diagnosis and/or for 
the determination of treatment parameters. 

Deformable models are a very general class of methods for the 
segmentation of structures in 3D images. Deformable models are known, e.g. from an 
article of T. Mclnerney et al. "Deformable models in medical image analysis: A 

25 survey 5 ', Medical Image Analysis, 1 (2):91-1 08, 1 996. 

The basic principle of deformable models consist of the adaptation of 
flexible meshes, represented, for example, by triangles or simplexes, to the object of 
interest in an image. For this, the model is initially placed near or on the object of 
interest in the image. This may be done by a user. Then, coordinates of surface 

30 elements of the flexible mesh, such as the triangles, are iteratively changed until they lie 
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on or close to the surface of the object of interest. Such a method is described in further 
detail in J. Weese et al. "Shape constrained deformable models for 3D medical image 
segmentation" in 17 th International Conference on Information Processing in Medical 
Imaging (IPMI), pages 380-387, Davies, CA, USA, 2001, Springer Verlag. 
5 Usually, several separate structures of interest, such as the femur head, 

the rectum, the prostate and the bladder in the prostate region, have to be segmented. 
Also, it often occurs that the structure of interest consists of several objects, such as two 
vertebra of the spine. In spite of having a certain spatial configuration, often, these 
objects are not connected. In such cases, adaptation may become a problem. 

10 If a plurality of obj ects are segmented one after the other, a correct 

segmentation of the individual objects is often not possible. It often occurs that the 
spatial relationship between the objects, for example, a minimum distance between two 
object boundaries cannot be maintained. 

It is also possible to carry out the segmentation process simultaneously 

15 for all objects. This can be done, for example, for reasons of speed and practicality. 
However, during a simultaneous segmentation of a plurality of objects, it often occurs 
that the spatial relationships are not maintained and are violated during mesh 
reconfiguration. This causes meshes which are intersecting or too close to each other, 
thus resulting in a wrong segmentation result. 

20 Accordingly, it is an object of the present invention to provide a method 

for the simultaneous segmentation of multiple or composed objects in an image which 
allows an improved simultaneous segmentation of multiple or composed objects. 

The above object is solved by a method for the simultaneous 
segmentation of multiple or composed objects in an image, wherein a deformable 

25 surface model is to be adapted to a first surface of a first object and a second surface of 
a second object and wherein the deformable surface model comprises a first partial 
deformable surface model and a second partial deformable surface model, comprising 
the steps of: 

(a) applying the first partial deformable surface model describing a 
30 structure of the first surface of the first object; 

(b) applying the second partial deformable surface model describing a 
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structure of the second surface of the second object, wherein the first partial deformable 
surface model and the second partial deformable surface model have a prescribed spatial 
relationship corresponding to a spatial relationship of the first object and the second 
object; and 

5 (c) adapting the first partial deformable model to the first surface and the 

second partial deformable model to the second surface, wherein the prescribed spatial 
relationship of the first partial deformable surface model and the second partial 
deformable surface model is used for the adaptation. 

In other words, in this method according to an exemplary embodiment 

10 according to the present invention, a priori knowledge of a spatial relationship of the 
first and second objects is used for adaptation. Advantageously, this method according 
to an exemplary embodiment of the present invention allows to carry out a simultaneous 
segmentation while maintaining spatial relationships between a plurality of objects 
during mesh configuration. Advantageously, this avoids intersecting meshes and/or 

1 5 meshes which are too close to each other. 

According to an exemplary embodiment of the present invention as set 
forth in claim 2, the step of applying a priori knowledge of the spatial relationship of 
the first and second objects for adaptation in accordance with step (c) of claim 1, 
involves the use of an additional edge connecting the surfaces of the two deformable 

20 surface models, or a plurality of additional edges which are connecting vertices of the 
two deformable surface models. The method according to claim 2 provides a simple 
methods whiqh can be executed with a minimized computation effort. Furthermore, the 
use of additional edges can be integrated without further amendments into other 
segmentation methods, such as the one known from J. Weese et al. 

25 According to another exemplary embodiment of the present invention as 

set forth in claim 3, the additional edge between objects is a featureless triangle. This 
means that assuming a mesh with triangles, an additional triangle is defined between a 
vertex of the first deformable surface model and a vertex of the second deformable 
surface model. The additional triangle is introduced such that when numbering the 

30 vertices of the triangle with 1, 2, 3, the vertices 1 and 3 of the additional triangle are- 
identical. Thus, the additional triangle does not have any surface. Accordingly, this 
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additional triangle does not contribute to a calculation of an external energy during 
segmentation. The motivation for no external energy (i.e. not searching for a surface 
feature) is that a connection between two deformable surface model does not 
correspond to an object surface. 
5 Further exemplary embodiments according to the present invention, as 

set forth in claims 4 and 5 provide, for a fast and efficient simultaneous segmentation of 
the two objects, while minimizing computation efforts. 

According to another exemplary embodiment of the present invention as 
set forth in claim 6, an extended internal energy relating to vectorial difference of 
10 additional edges and approximately a distance between the two objects is introduced 
into the segmentation^ A minimization of the internal energy with the extended internal 
energy allows to keep the spatial relationships of the two objects. 

Claim 7 relates to an exemplary embodiment of an image processing 
device suitably adapted for executing the method according to the present invention. 
15 Advantageously, this image processing device allows a very accurate simultaneous 
segmentation while avoiding intersecting meshes and while avoiding a violation of 
spatial relationships of the objects. 

Claim 8 relates to an exemplary embodiment of a computer program 
executing the method according to the present invention. Advantageously, this 
20 computer program allows for a simultaneous segmentation of multiple or composed 
objects, without the occurrence of intersecting meshes or a violation of spatial 
relationships between the objects. 

It is the gist of the present invention, to include a priori knowledge 
concerning a spatial relationship of the two objects into the segmentation. For this, 
25 additional connections are introduced between vertices into a shape model which only 
contribute to the internal energy, but are not used to detect feature points and therefore 
do not contribute to the external energies. If, for example, the distance between two 
surfaces is approximately known, additional connections between these surfaces are 
introduced. Maintaining the length of these connections (i.e. minimizing an extended 
30 internal energy) will minimize the deviation of the original distance between the 
objects. 
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These and other aspects of the present invention will become apparent 
from and elucidated with reference to the embodiments described hereinafter. 

Exemplary embodiments of the present invention will be described in the 
following, with reference to the following drawings: 
5 Fig. 1 shows a schematic representation of an image processing device 

according to an exemplary embodiment of the present invention adapted to 
execute a method according to an exemplary embodiment of the present 
invention. 

Fig. 2 shows a flow chart of an exemplary embodiment of a method 
10 according to the present invention. 

Fig. 3 shows a triangular surface model of two vertebrae of the spine for 
explaining an exemplary embodiment of the present invention. 

Fig. 4 shows two vertebrae with additional connections between upper 
and lower vertebra for explaining an exemplary embodiment of the present 
15 invention. 

Fig. 5 shows a simplified drawing for explaining a featureless vertex 
connection in accordance with an exemplary embodiment of the present 
invention. 

Figure 1 shows a simplified schematic representation of an exemplary 
20 embodiment of an image processing device in accordance with the present invention. In 
Fig. 1, there is shown a central processing. unit (CPU) or image processor 1 for adapting 
a deformable model surface to surfaces of multiple or composed objects by mesh 
adaptation. Preferably, this adaptation is performed simultaneously for a plurality of 
objects in an image. The image processor 1 is connected to a memory 2 for storing the 
25 deformable surface model comprising a first and a second partial deformable surface 
model, and for storing the image depicting the objects of interest. The image processor 
1 may be connected via a bus system 3 to a plurality of periphery devices or 
input/output devices which are not depicted in Fig. 1 . For example, the image processor 
1 may be connected to a MR device, a CT device, to a plotter or a printer or the like via 
30 bus system 3. Furthermore, the image processor 1 is connected to a display such as a 
computer screen 4 for outputting segmentation results or information. Furthermore, a 
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keyboard 5 is provided, connected to the image processor 1, by which a user or operator 
may interact with the image processor 1 or may input data necessary or desired for the 
segmentation process. 

Figure 2 shows a flow chart of a method or computer program according 
5 to an exemplary embodiment of the present invention which is executed on an image 
processing device shown in Figure 1. 

After initialisation in step 100, in step 101, a deformable model M and an 
image I containing a structure of interest are loaded from the memory 2 into the image 
processor 1. Then, the model is coarsely placed over the object of interest in the image 

10 I. This can be done, e.g. by a user using the keyboard 5, a mouse or a similar interactive 
device, or a suitable automated initial placement method as known in the art. 

The composed deformable model M represents surfaces of several 
anatomical objects and their spatial constellation. Fig. 3 shows an example of the 
surface model of two vertebrae. Here, the deformable surface model M comprises the 

1 5 first partial deformable surface model to be adapted to the surface of the first vertebra 
and the second partial deformable surface model to be adapted to the surface of the 
second vertebra. In spite of the fact that two vertebrae are depicted, other structures, like 
e.g. the femur and the hip, or the inner or outer wall of the heart ventricle are possible. 
In case, more than two objects are to be segmented simultaneously, a number of partial 

20 deformable surface models may be provided. 

The depicted composed first and second partial deformable models 
consist of the surfaces of the two vertebrae. Each object is represented by a polygonal 
mesh. In the present example, vertices Xi are connected by triangles tj. Nevertheless, 
other representations, such as simplex or general polygonal meshes are possible. Such a 

25 surface representation can be derived by a triangulation method of a single training 
object, as described, for example, in W.E. Lorensen, et al. "Marching cubes: A high 
resolution 3D surface construction algorithm" Computer Graphics, 21(3): 163-169, 
1987. 

In Figure 4, according to an exemplary embodiment of the present 
30 invention, a priori knowledge of the spatial relationship of the two vertebrae is included 
in the model. As shown in Fig. 4, the model was augmented with additional edges 
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which connect vertices of the first vertebra with vertices on the second vertebra. In Fig. 
4, these additional dedicated connections between neighbouring triangles of the meshes 
of the two vertebrae are clearly visible between, e.g. the top endplate of the lower 
vertebra and the lower endplate of the top vertebra. These dedicated connections 

5 represent a priori knowledge of the spatial relationships between the objects. These 
connections between vertices of the neighbouring vertebrae are derived as follows: 
Given two objects Oi and O2 (the two vertebrae) each with a set of 
vertices x u and . For each vertex xn of object Oi, the closest vertex x a of object 0 2 is 
searched. This results in a list of vertex pairs. {( x oi, xi k i), (*n 5 x ik2>, . . }• For each 

10 vertex pair, a length | xoi - xiki | is calculated and only those pairs with a length smaller 
than a certain threshold T are kept. This results in the fact, as shown in Fig. 4, that 
connections are only introduced between vertices which are not too far distanced from 
each other. As can be taken from Figure 4, with the adjustment of the threshold T, 
connections are only introduced between neighbouring triangles of the two vertebrae. 

1 5 The resulting pairs are stored in the model as additional or "connection" triangles c k , 
where point 0 and point 2 are the vertex xoi of object Oi, and point 1 of the triangle is 
the corresponding vertex xi k i of object 0 2 . This is depicted in further detail in Figure 5. 

In Figure 5, there is a first triangle 10 on the surface of object Oi and a 
20 second triangle 20 on the surface of the object 0 2 . For the sake of clarity, only one 

triangle of the mesh is depicted for each object Oi and 0 2 . As can be taken from Fig. 5, 
the vertex xoi on the surface of object Oi is connected with the vertex x^i on the surface 
ofobject0 2 . This connection is the additional or the "connection" triangle c^ The 
triangle 20 shows a numbering of the comers of the triangle 20. The comer of the 
25 triangle 20 of the vertex x^i is designated with 0', the lower comer on the left side is 
designated with l f and the lower comer on the right side of the triangle 20 is designated 
with 2\ Using this way of numbering the comers of the triangle, it can be taken from 
Figure 5 that the comers 0 and 2 of the triangle c k are the vertex xoi and the comer 1 of 
the triangle c k is the vertex xi k i. Due to this, the triangle has no surface. In other 
30 words, the connection triangle c k is a degenerated triangle, meaning that the edge 
between points 0 and 2 has a zero length, and the edge between points 0 and 1 is 
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identical with the edge between points 2 and 1. Also, the edges between 0 and 1, 
respectively, 2 and 1 are at the same distance as the vertices Xoi and xm\ Hence, in 
total, the model consists of a set of surface voxels {x;}, a set of surface triangles {tj}, 
and a set of connection triangles {Ck}. Accordingly, as may be taken form Figure 4, in 
5 the deformable model, additional connection triangles between the upper and the lower 
vertebra were added to the surface triangles. 

Having generated the composed deformable model and initialised the 
starting position in the image, an iterative adaptation procedure is carried out in steps 
102 and 103. Here, the first and second partial deformable surface models are adapted 
10 to the first and second objects, i.e. the two vertebrae. The procedure iteratively varies 
the vertices coordinates so that they optimally fit the surface of the structure of interest, 
i.e. the two vertebrae. 

To adapt the deformable surface model to the object in the image, the 
method of T. Mclnterney et al. may be used. However, preferably a modified method 
1 5 based on Weese et al. is used. For a better understanding, the modified method by 

Weese et al. is briefly summarized for single objects, and then the extension to multiple 
objects with connection triangles is explained. 

Given an initial surface model of the object, each iteration consists of a 
feature detection step (step 102) and a mesh reconfiguration step (step 103). In step 
20 102, for each of the triangles, a search along the triangle normal is carried out to find a 
point jc i that potentially lies on the boundary of the object in the image. 

In step 103, the position of the mesh vertices are recalculated by energy 
minimization with respect to the vertex coordinates. The energy is a weighted sum of 
two terms, 

25 E-E^+aEinr (J) 

The external energy E ext drives the mesh vertices towards the surface 
elements associated with the detected feature points and can be calculated as in Weese 
etal. 

The internal energy restricts the movement of the mesh vertices. Since 
30 the distribution of vertices on the initial template mesh is optimal with respect to 
surface-curvature, advantageously that distribution is maintained by punishing the 
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change of distance between neighbouring vertices. Therefore, it is defined: 
N V t 

E iM =Z Z Z(%-^-^) 2 ' (2) 

i MMM/O') 

where N is the number of objects, U is the set of vertices belonging to the 
i-th object, Vj is the number of vertices of the i-th object, x k and x j represent two 
5 neighboring vertices of the deformable mesh as found in step 102, and Aj k is the 

corresponding vectorial difference between vertex j and its neighbour k of a reference 
model. A scaling s and a rotation matrix R between the initial model and the deformed 
model are also estimated. 

The result of this minimization are new vertex coordinates, which reflect 
10 a trade-off between being close to the surface points detected in the image and staying 
similar to the initial deformable model. From these new vertex coordinates new 
boundary points are searched for in step 102, and so forth. These two steps are iterated 
.a fixed number of times, leading to a deformed model close to the surface of the object 
of interest. 

15 In the following, the method of Weese et al. will be extended for the 

adaptation of composed deformable models. If the structure of interest is composed of 
several objects which are not connected but spatially related in a certain way, correct 
adaptation may become a problem. The spatial relationships may be violated during 
mesh reconfiguration, leading to meshes which are intersecting or too clo§e to each 

20 other. One of the main reasons is that no information about the spatial coupling of the 
vertices of the different meshes is taken into account during adaptation in the known 
methods. 

According to an exemplary embodiment of the present invention, in 
order to maintain certain spatial relationships between the meshes, "connections 
25 triangles" are introduced, in order to approximately maintain the distance between the 
two objects, or referring to the examples of Figs. 3 and 4, to maintain the spatial 
relationships between the two vertebrae. As may be seen from the following, the spatial 
relationship is used during adaptation. 

As described above, according to an exemplary embodiment of the 
30 present invention, additional connection triangles cr, each of which connects a vertex of 
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the lower vertebra to a vertex of the upper vertebra, are introduced. In order to 
approximately maintain the initial distances between the vertebrae, preferably, the initial 
configuration of the connection triangles is maintained. This can be done by adding an 
additional energy (see equation (1)), i.e. 

E = i?ext "HxCEint + ^connections) 

The purpose of 2?connections is to restrict the change of the connection 
triangles. The connection energy can be formulated as 



Econnections : 

(4) 



Z I X(*, -%-^*) 2 . 



Here, only those vertices are considered which belong to a set of 
connection triangles c*. Accordingly, only those vertex neighbours N c (j) are considered, 
which belong to the same connection triangle as the vertex Xj. Ajk is the vectorial 
distance between two points of a connection triangle. 

It is important that only triangles are introduced, but no additional 
vertices (note that this it true because the additional triangles were generated from the 
available surface vertices). The internal energy E mt and the connection energy ^connection 
are combined into one internal energy 

N V, 

This leads to the new energy function of the composed deformable model 
E = E m +a£ M (6) 

Then, the energy is minimized. Here it should be noted again that due to 
the fact that the connection triangles cr do not have a surface, no feature search is 
carried out in step 102 for the connection triangles Ck. Thus, a feature search is only 
carried out for the surface triangles tj. This advantageously allows to minimiz e the 
computation efforts. 

Then, for displaying the deformable model, the vertices x,- and the 
triangles tj can be selected in step 104 and displayed on the computer screen 4. 
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However, it should be noted that instead of displaying the entire deformed model, an 
output of the vertex coordinate numerically is also possible. Then, the method 
continues to step 105, where the method ends. 

In brief, according to the present invention, the spatial relationship of 
5 objects is used during adaptation of the deformable surface model to the objects. 

Accordingly, due to the inclusion of additional connections to the vertices into the shape 
model, which vertices only contribute to the internal energy, but are not used to detect 
feature points, and therefore do not contribute to the external energies, spatial 
relationships between multiple or composed objects can be maintained very easily by 

10 minimizing the internal energy. This allows to accurately maintain spatial relationships 
between objects which are simultaneously segmented. In particular, the danger is 
prevented that spatial relationships between objections are violated and that the 
composed deformable models intersect each other. A segmentation of multiple objects 
at the same time also improves the robustness and speed of segmentation using 

1 5 deformable models. 



